Scrap approval of test results, introduce color scale instead#1221
Conversation
|
I took the liberty to deploy this to the compliance monitor (after I tested it locally, of course). I found a defect, that I resolved. Now it should be good. Have a look!
BTW the views use the same color scale; the only difference is that the former shows draft versions. This is the color scale: COLOR_MAP = {
-1: '🛑', # fail
None: '🟧', # missing
0: '✅*', # inconclusive
1: '✅', # pass
}I think the asterisk still needs to be explained, and it collides with the asterisk for draft versions. I'm sure we will get this ironed out. |
|
So, right now, with scs2, we have three testcases that couldn't be tested properly: 
(the virtual machine didn't come up in time) Which means: we don't know whether these testcases are actually satisfied or not. Therefore, we now get the asterisk in the summary: 
|
|
Note: the asterisk for "draft" can probably just be dropped, because a) we have the convention to call the draft "next", which is quite telling already, The double dagger for "deprecated" can probably be removed as well, because we have a convention to not show deprecated versions. What remains is to put the legend of symbols at the top, and presto. |
|
After #1222, some subjects are missing the result for scs-0114-syntax-check, and so we see this: 
|
|
Now the Sonobuoy tests on Noris take so long that they literally time out. Luckily, the reports got sent anyway (apparently, Zuul is kind of gentle with killing the process; I don't know how this works). Notice the unfinished test cases in the tables and the corresponding asterisks in the summary at the top 
|
Show failed (some required testcase failed) as red, missing (some required testcase missing) as yellow, inconclusive (some required testcase aborted) as green with asterisk, passed (all required testcases passed) as green. The reduces the probability of confusing people with false positives, while no longer requiring (unrealistically) labor-intensive approvals. Besides, the two views 'table_full' vs 'table' now only differ in that the former shows draft versions; similarly for 'details[_full]'. Further work remains: - remove approval column from database - remove approval-related code from sql.py - remove endpoint for POSTing /results - simplify, consolidate, isolate business logic - etc. We're doing this incrementally... for now, let's see whether this works at all. Signed-off-by: Matthias Büchse <matthias.buechse@alasca.cloud>
Signed-off-by: Matthias Büchse <matthias.buechse@alasca.cloud>
2 participants
Show failed (some required testcase failed) as red, missing (some required testcase missing) as yellow, inconclusive (some required testcase aborted) as green with asterisk, passed (all required testcases passed) as green.
The reduces the probability of confusing people with false positives, while no longer requiring (unrealistically) labor-intensive approvals. Besides, the two views 'table_full' vs 'table' now only differ in that the former shows draft versions; similarly for 'details[_full]'.
Further work remains:
We're doing this incrementally... for now, let's see whether this works at all.